1. Field of the Invention
The present invention relates to a confocal optical scanner configured to acquire rapidly (in real time) super resolution images where influence of spurious resolution and artifacts due to image processing is inhibited.
Priority is claimed on Japanese Patent Application No. 2014-027918, filed Feb. 17, 2014, the contents of which are incorporated herein by reference.
2. Description of Related Art
Hereinafter, three related arts in the technical field of confocal optical scanner will be described with reference to drawings.
(A) Related Art 1
A technique for acquiring images of a specimen, which have a resolution higher than the resolution limit of optical system (Abbe diffraction limit), has been developed and put to practical use. Hereinafter, such a technique is referred to as a super resolution technique. An example of super resolution technique includes a technique described in Japanese Patent Application Laid-Open Publication No. 2012-78408 (Japanese Patent No. 5412394).
(A1) Configuration and Operation of Related Art 1
FIG. 16 is a schematic diagram showing one configuration example of confocal optical scanner in Related Art 1. FIG. 16 shows a fourth embodiment (FIG. 10A) described in Japanese Patent Application Laid-Open Publication No. 2012-78408 (Japanese Patent No. 5412394). Hereinafter, the configuration and operation of confocal optical scanner in Related Art 1 will be described with reference to FIG. 16.
A confocal optical scanner 100 includes a microlens disk 102, on which a plurality of microlenses 102a is regularly disposed, a pinhole disk 103, on which pinholes 103a are disposed, and a motor 104 for rotating the microlens disk 102 and the pinhole disk 103. Each pinhole 103a is an opening of a light shielding mask 103b and is positioned opposite to the microlens 102a. 
A light source device 105 includes a light source such as a laser and an optical system, which are not shown, and is configured to output collimated illumination light. The illumination light input into the confocal optical scanner 100 is divided into a plurality of illumination light beamlets by the plurality of microlenses 102a disposed on the microlens disk 102. The divided illumination light is transmitted through a beam splitter 106 and passes through the pinhole positioned opposite to the microlens 102a, through which the illumination light has been passed, among the plurality of pinholes 103a disposed on the pinhole disk 103. In order to make the illumination light be passed through each pinhole 103a, each pinhole 103a is disposed on the focal plane of the microlens 102a. 
The illumination light, which has been passed through the pinhole disk 103, is condensed onto a specimen 108 by an objective lens 107. The specimen 108 outputs return light based on the illumination light. In particular, in a case of observation of a fluorescent specimen, the specimen 108 is stained using a fluorescent dye so as to have a specific structure. The fluorescent dye molecule of the specimen 108 is excited by the illumination light and the specimen 108 outputs fluorescence having a longer wavelength than the illumination light.
The return light captured by the objective lens 107 is condensed onto the pinhole disk 103 provided in the confocal optical scanner 100. At this time, only return light from the focal plane of the objective lens 107 facing the specimen passes through the pinhole 103a. On the other hand, since return light from other than the focal plane is not focused on the pinhole 103a and is shielded by the light shielding mask 103b disposed on the pinhole disk 103, most of the return light cannot pass through the pinhole 103a. 
The return light, which has passed through the pinhole 103a, is reflected by the beam splitter 106. In particular, in a case of fluorescent observation, the beam splitter 106 is for dispersing light based on a wavelength and has a short pass characteristic where illumination light is transmitted and return light, which is fluorescence and has a longer wavelength than the illumination light, is reflected. The return light reflected by the beam splitter 106 forms an image on a camera 110 by an imaging lens 109.
At the same time, the microlens disk 102 and the pinhole disk 103 are rotated by the motor 104, and the whole of specimen 108 is scanned using illumination light. This enables a confocal image (optical cross-sectional image) of the specimen 108 to be imaged using the camera 110.
At this time, the illumination light, which has a spatial intensity distribution modulated by the pinhole pattern of the light shielding mask 103b, is projected on the specimen 108. Thereby, in the return light from the specimen 108, a part of high-frequency component beyond a resolution limit of optical system is shifted to a frequency below the resolution limit. In addition, by adopting the configuration where the return light passes through the pinhole pattern of the light shielding mask 103b, the shifted band is demodulated into the original high-frequency component. Therefore, a confocal image having a high-frequency component beyond a resolution limit of optical system is imaged by the camera 110. Since the high-frequency component beyond a resolution limit of optical system has low contrast compared to a low-frequency component and cannot be sufficiently visualized as an image, the high-frequency component is subjected to a high-frequency enhancement process using an image processing board 111 and a personal computer 112. Therefore, a confocal image where a high-frequency component beyond a resolution limit of optical system is sufficiently visible can be obtained.
(A2) Problems in Related Art 1
As described above, in order to obtain a confocal image where a high-frequency component beyond a resolution limit of optical system is sufficiently visible, it is necessary to subject a confocal image imaged by a camera to a high-frequency enhancement process. Therefore, various spurious resolution and artifacts occur due to noise components included in a confocal image imaged by a camera.
In the field of natural science for observing “nature” (for example, a field for observing a biological specimen, a cell, and the like using a microscope), there are some cases where “artifacts (data distortion and errors occurred in an observation and analysis process, and the like)” occur.
Since parameters such as strength and a band in the high-frequency enhancement process are not obvious, it is necessary to determine parameters by trial and error for each image so as to prevent occurrence of spurious resolution and artifacts in images obtained by performing the high-frequency enhancement process. In addition, it is impossible to determine whether the high-frequency components, which are visualized as a result of the process, are generated based on the microscopic structure of actual specimen or are generated due to spurious resolution.
The high-frequency enhancement process improves the resolution of image in the imaging plane (X-Y plane), but does not improve the resolution of image in the light axis direction (Z-axis direction) perpendicular to the image. Therefore, there are some cases where it is not easy to observe in detail the spatial structure of specimen.
Since the high-frequency enhancement process requires a long processing time, there are some cases where it is not easy to display super resolution images in real time. In addition, since the high-frequency enhancement process requires a personal computer with high performance and an image processing board, there are some cases where the device configuration is complicated and expensive.
(B) Related Art 2
Examples of confocal microscope having a super-resolution effect include, for example, an “Image Scanning Microscopy (ISM)” method described in Schulz, O. et al. Resolution doubling in fluorescence microscopy with confocal spinning-disk image scanning microscopy, Proceedings of the National Academy of Sciences of United States of America, Vol. 110, pp. 21000-21005 (2013). Hereinafter, the configuration and operations of the confocal microscope will be described with reference to FIG. 17.
(B1) Configuration and Operation of Related Art 2
FIG. 17 is a schematic diagram showing one configuration example of confocal microscope in the related art. Hereinafter, the configuration and operation of the confocal microscope will be described with reference to FIG. 17.
An ISM confocal microscope uses a similar confocal optical scanner to that of Related Art 1. A confocal optical scanner 200 includes a shutter 205a in a light source device 205. In addition, the confocal optical scanner 200 includes a motor 204 and a synchronization controller 213 for synchronizing the shutter 205a and a camera 210.
The shutter 205a allows illumination light from the light source device 205 to pass through for only a short time, a few microseconds at each photographing and outputs stroboscopic light. In the short illumination period, the pinhole 203 rotated by the motor 204 is considered to have a stopped state. Therefore, by adopting the configuration where the camera 210 performs an imaging for a short time, which is the same as the illumination period, confocal images in the nearly stopped state (non-scanning confocal image) can be obtained without scanning a specimen 208 with illumination light. Since the non-scanning confocal image is generated by imaging only return light from the positions on the specimen 208, which are opposite to a plurality of pinholes 203a in the pinhole pattern of a light shielding mask 203b, a plurality of bright points is recorded in one image.
A few hundred non-scanning confocal images as described above are imaged with performing a synchronization control of the motor 204, the shutter 205a, and the camera 210 using the synchronization controller 213.
When the non-scanning confocal images are imaged, the synchronization control is performed so that the opening-closing timing of the shutter 205a for the rotation of the motor 204 is deviated at a regular interval. Thereby, the position of each of the bright points is slightly different in each image and the whole of image is filled with the bright points by superimposing all images.
The following image processing is performed for a few hundred non-scanning confocal images obtained as described above. The central coordinate of each of the plurality of bright points recorded in the non-scanning confocal image is calculated, and pixels near the bright point are shifted so that the distance to the center is half of the original distance. In other words, the image processing for decreasing the size of each of the plurality of bright points so as to be half of the original size is performed. Finally, a super resolution image is obtained by superimposing a few hundred non-scanning confocal images subjected to the bright point decreasing process.
Hereinafter, reasons for acquisition of super resolution images will be described with reference to FIGS. 18 and 19. FIG. 18 is a schematic diagram showing a confocal optical system using a two-dimensional image sensor (camera). In order to simplify the diagram, an illumination side and an imaging side are separately shown. The illumination side indicates a region from a point light source to a specimen plane, and the imaging side indicates a region from the specimen plane to an imaging plane. In order to simplify the diagram, the magnification of objective lens is set to 1×, but an objective lens having a magnification other than 1× may be used. When the magnification of objective lens is set to 1×, the specimen plane and the imaging plane are the same in the scale as each other and are opposite in the scale direction to each other.
The illumination light output from the point light source on the optical axis is focused on the specimen plane by the objective lens. At this time, the light diffraction causes the intensity distribution of illumination light on the specimen plane to have a certain extent centered at the coordinate x=zero as shown in FIG. 18. The extent of light is generally referred to as Airy disc. Next, return light, which is generated by exposing the specimen to the illumination light and is output from three points at the coordinates, x=zero, d/2 and d on the specimen plane, will be considered. A description will be provided for the case where these three points exist in the Airy disc of the illumination light. In FIG. 18, first to third curved lines are shown in front of the imaging plane. The first curved line corresponds to the intensity distribution curved line of return light output from the coordinate x=zero on the specimen plane. The second curved line corresponds to the intensity distribution curved line of return light output from the coordinate x=d/2 on the specimen plane. The third curved line corresponds to the intensity distribution curved line of return light output from the coordinate x=d on the specimen plane. As shown in FIG. 18, the first intensity distribution curved line has the peak at the coordinate x=zero on the imaging plane, and the peak height is proportional to the illumination light intensity at the coordinate x=zero on the specimen plane. The second intensity distribution curved line has the peak at the coordinate x=d/2 on the imaging plane, and the peak height is proportional to the illumination light intensity at the coordinate x=d/2 on the specimen plane. The third intensity distribution curved line has the peak at the coordinate x=d on the imaging plane, and the peak height is proportional to the illumination light intensity at the coordinate x=d on the specimen plane.
The return light is received by a two-dimensional image sensor (camera) provided on the imaging plane. The amount of light received at the position of the coordinate x=d on the imaging plane (the position corresponding to Pixel 2 shown in FIG. 18) will be considered. In FIG. 18, the intensity distributions of return light output from the coordinates x=zero, d/2, and d on the specimen are compared to one another at the coordinate x=d on the imaging plane. This shows that the intensity of the return light from the coordinate x=d/2 on the specimen plane is the largest. In other words, the pixel at the position of the coordinate x=d on the imaging plane receives the brightest light which is not return light from the coordinate x=d on the specimen plane, but return light from the coordinate x=d/2 on the specimen plane. This shows that, in the microscopic area of the Airy disc, the distribution on the specimen plane is increased to twice and projected on the imaging plane.
The above-described optical phenomenon is explained using equations as follows. When a position on specimen plane where return light occurs is defines as x and an amount of light received at a position on imaging plane d is defined as I(x), the amount of light received I(x) is represented by the following equation (1).
Where, PSFill(x) and PSFimg(x) are a point spread function on the illumination side and a point spread function on the imaging side, respectively.I(x)=PSFill(x)×PSFimg(x−d)  (1)
Generally, the point spread function PSF(x) is represented by the following equation (2) using Bessel function of the first kind J1, and a numerical aperture NA and a wavelength λ of an optical system.
                              PSF          ⁡                      (            x            )                          =                              (                                                            J                  1                                (                                  2                  ⁢                                      π                    ·                    NA                    ·                    x                                    ⁢                                      /                                    ⁢                  λ                                                                              π                  ·                  NA                  ·                  x                                ⁢                                  /                                ⁢                λ                                      )                    2                                    (        2        )            
According to the equation (1), I(x) is represented as a product of two point spread functions where the peak position of one of the point spread functions is different from that of the other by a distance d. Therefore, the sketch of I(x) has a peak at d/2 as shown in FIG. 19 (horizontal axis: coordinate, vertical axis: light intensity). In other words, the equation (1) also indicates that the pixel at the position of the coordinate x=d on the imaging plane receives the brightest light which is return light from the coordinate x=d/2 on the specimen plane.
As described above, in the confocal optical system using the two-dimensional image sensor (camera), the distribution of images on the specimen plane is increased to twice and projected on the imaging plane in the region of the Airy disc centered each bright point of the non-scanning confocal image. Thereby, by reducing the distribution of images in the Airy disc to half and performing a correction process for conforming the coordinate on the specimen plane to the coordinate on the imaging plane, the high-frequency component beyond a resolution limit of optical system can be obtained. The reason is that the process for reducing the distribution of images in the Airy disc to half corresponds to a process for reducing the width of the point spread function of the optical system to half. Thereby, super resolution images having a resolution, which is twice as large as the resolution limit (diffraction limit) of an optical system, can be obtained.
(B2) Problems in Related Art 2
As described above, the technique in Related Art 2 requires imaging of a few hundred non-scanning confocal images for obtaining one super resolution image. At this time, since the imaging of the one super resolution image requires several tens of seconds, the time resolution is low and it is not easy to capture a rapid phenomenon. Therefore, there are some cases where it is difficult to display super resolution images in real time.
In addition, since the technique in Related Art 2 requires a rapid shutter, a synchronization control device, and a personal computer with high performance, there are some cases where the device configuration is complicated and expensive.
In addition, the technique in Related Art 2 improves the resolution of image in the imaging plane (X-Y plane), but does not improve the resolution of image in the light axis direction (Z-axis direction) perpendicular to the image. Therefore, there are some cases where it is not easy to observe in detail the spatial structure of specimen.
(C) Related Art 3
Other Examples of confocal microscope having a super-resolution effect include, for example, a “Multi-Focal Structured Illumination Microscopy” method described in WO 2013/126762. Hereinafter, the configuration and operation of the “Multi-Focal Structured Illumination Microscopy” method will be described with reference to FIG. 20.
(C1) Configuration and Operation of Related Art 3
A confocal optical scanner 350 includes microlens arrays 341, 352 and 353, a pinhole array 351, a galvanic mirror 349, a beam splitter 306, relay lenses 343, 344 and 345, and mirrors 346 and 347. In each of the microlens arrays 341, 352 and 353, a plurality of microlenses is regularly disposed.
The pinhole array 351 includes a plurality of pinholes 351a. Each pinhole 351a is disposed on a position, which optically corresponds to (is conjugate to) the focal position of each microlens 341a included in the microlens array 341. The pinhole 351a is an opening of a light shielding mask 351b. The microlens array 352 includes a plurality of microlenses 352a. Each microlens 352a is disposed on a position, which corresponds to each pinhole 351a included in the pinhole array 351. In addition, the microlens array 353 includes a plurality of microlenses 353a. Each microlens 353a is disposed on a position, which corresponds to each microlens 352a included in the microlens array 352. Each microlens 341a, 352a, and 353a may be replaced by another optical element (for example, a Fresnel lens and a diffractive-optical element) as long as the another optical element has a lens effect.
The interval between the pinhole array 351 and the microlens array 352 is equal to the focal length of each microlens 352a included in the microlens array 352. Therefore, the microlens array 352 converts the beam input from the side facing the pinhole array 351 so that the converted light is parallel light in the space where the converted light output from the microlens array 352 exists. The focal length of each microlens 353a included in the microlens array 353 is set to half of the focal length of the microlens 352a. Therefore, the microlens arrays 352 and 353 convert the beam input from the pinhole array 351 so that the converted beam has a numerical aperture which is twice as large as that of the light before being input into the microlens array 352 in the space where the converted beam output from the microlens array 353 exists.
A light source device 305 includes a light source such as a laser and an optical system, which are not shown, and outputs collimated illumination light. The illumination light is divided into a plurality of illumination light beamlets by the microlens array 341. The microlens array 341 may be designed so that the numerical aperture of the illumination light beamlet is close to or greater than a value obtained by dividing a numerical aperture of an objective lens 307 by the magnification.
The illumination light passes through the beam splitter 306 and the relay lens 343, is reflected by the galvanic mirror 349, passes through the relay lens 344 and the objective lens 307, and is condensed onto a specimen 308. At this time, by varying the direction of the surface of the galvanic mirror 349, the whole of specimen 308 is scanned using the illumination light.
The specimen 308 outputs return light based on the illumination light. In particular, in a case of observation of a fluorescent specimen, the specimen 308 is stained using a fluorescent dye so as to have a specific structure. The fluorescent dye molecule is excited by the illumination light and the specimen 308 outputs fluorescence having a longer wavelength than the illumination light.
The return light captured by the objective lens 307 passes through the relay lens 344, is reflected (descanned) by the galvanic mirror 349, passes through the relay lens 343, and is reflected by the beam splitter 306. In particular, in a case of fluorescent observation, the beam splitter 306 is for dispersing light based on a wavelength and has a short pass characteristic where illumination light is transmitted and return light, which is fluorescence and has a longer wavelength than the illumination light, is reflected.
The return light reflected by the beam splitter 306 is focused on the pinhole array 351 and passes through the pinhole 351a. At this time, only return light from the focal plane of the objective lens 307 facing the specimen passes through the pinhole 351a. On the other hand, since return light from other than the focal plane is not focused on the pinhole 351a and is shielded by the light shielding mask 351b included in the pinhole array 351, most of the return light cannot pass through the pinhole 351a. 
The return light, which has passed through the pinhole 351a, is converted into beam having a numerical aperture which is twice as large as that of the light before being input into the microlens array 352, by the microlens arrays 352 and 353.
The return light, which has passed through the microlens array 353, passes through the relay lens 345, the mirror 346, and the mirror 347, is reflected (rescanned) by the galvanic mirror 349, and is focused on a camera 310 by an imaging lens 348. At this time, the numerical aperture of each of the relay lens 345 and the imaging lens 348 may be close to or greater than that of the return light, which has been converted by the microlens arrays 352 and 353 so as to have the numerical aperture which is twice as large as that of the light before being input into the microlens array 352.
At the same time, the direction of the surface of the galvanic mirror 349 is varied to scan the whole of specimen 308 using the illumination light and the return light from the specimen 308 is scanned and projected on the camera 310. This enables the super resolution confocal images of the specimen 308 to be imaged using the camera 310.
As described above with reference to FIGS. 18 and 19, in the “Related Art 2”, in the confocal optical system using the two dimensional image sensor (camera), the distribution of images on the specimen plane is increased to twice and projected on the imaging plane in the region of the Airy disc centered each bright point. Therefore, by reducing the distribution of images in the Airy disc to half and performing a correction process for conforming the coordinates on the specimen plane to the coordinates on the imaging plane, the high-frequency component beyond the resolution limit of optical system can be obtained. On the other hand, in the “Related Art 3”, the reduction of the distribution of images in the Airy disc to half is optically performed. The reason is that, by increasing the numerical aperture of the return light from the objective lens 307 to twice as large as before by the microlens arrays 352 and 353, the width of the point spread function of the optical system becomes half in accordance with the equation (2), in other words, the distribution of images in the Airy disc, which are reduced to half, is projected on the camera 310.
In addition, in the “Related Art 2”, it is necessary to image a few hundred non-scanning confocal images and to integrate them. On the other hand, according to the “Related Art 3”, since the distribution of images in the Airy disc is optically reduced to half and the reduced distribution is imaged by the camera 310, it is only necessary to perform one imaging during the scanning of the whole of the specimen 308 with varying the direction of the surface of the galvanic mirror 349. Therefore, super resolution images having a resolution, which is twice as large as the resolution limit (diffraction limit) of optical system, can be easily obtained in a short time.
(C2) Problems in Related Art 3
In order to implement the Related Art 3, it is important to stably ensure the following three points.                It is necessary to precisely dispose the microlenses 341a included in the microlens array 341 and the pinholes 351a included in the pinhole array 351 so that the focal position of each microlens 341a included in the microlens array 341 optically corresponds to (is conjugate to) the position of each pinhole 351a included in the pinhole array 351.        It is necessary to precisely dispose each pinhole 351a included in the pinhole array 351 on the focal position of each microlens 352a included in the microlens array 352.        It is necessary to precisely dispose each microlens 353a included in the microlens array 353 so that each microlens 353a has the same axis as the microlens 352a included in the microlens array 352.        
As described above, it is necessary to precisely dispose each of a plurality of micro optical elements (the microlens arrays 341, 352 and 353, and the pinhole array 351). The micro optical elements are spatially separated from one another. Therefore, there are some cases where the Related Art 3 requires a plurality of precise position and angle adjustment mechanisms, the configuration is complicated and expensive, and the optical adjustment is not easy. In addition, in the Related Art 3, since the micro optical elements are spatially separated from one another, there are some cases where the relative position among the micro optical elements is changed due to a changing in circumstances such as a temperature, the optical adjustment easily collapses, and the micro optical elements cannot be stably used in a long time.